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ABSTRACT 

This  report  describes  and  gives  the  results  of  an  experiment  designed 
to  investigate  sequencing  constructs  in  computer-aided  instruction  (CAl) 
programs.   The  experiment  compared  two  languages.   One  incorporated  sequencing 
constructs  of  an  established  CAI  programming  language;  the  other  contained 
alternate  constructs  which  stressed  uniformity,  orthogonality,  and  static 
definition  and  minimized  the  use  of  hidden  side  effects. 

Working  on-line,  subjects  debugged  and  modified  a  large  program  while 
the  system  monitored  their  progress  and  gathered  data.   Although  several 
interesting  results  emerged,  it  was  impossible  to  state  conclusively  that 
the  entire  set  of  sequencing  constructs  in  either  language  proved  better. 
Instead,  the  experiment  indicated  that  direct,  explicit,  and  simple 
constructs  are  best.   Recommendations  are  given  for  designing  sequencing 
features  for  CAI  languages. 
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1.   INTRODUCTION 

1.1  Psychological  Studies  in  Programming 

Programming  languages  have  traditionally  teen  designed  by  individuals 
or  relatively  small  groups.   Language  designers  generally  introduced  and 
propagated  language   features  based  on  their  own  feelings,  thoughts,  and 
experiences.   They  rarely  consulted  with  the  intended  user  community  and  made 
little  or  no  effort  to  determine  the  psychological  soundness  of  newly  designed 
language  constructs.   One  objective  method  to  consult  the  wishes  of  the  user 
community  and  concurrently  establish  psychological  soundness  of  new  language 
constructs  is  through  thorough,  carefully  documented,  and  replicable  experiments 
[l].   Through  empirical  tests,  designers  can  gain  assurance  that  their  language 
features  actually  meet  the  needs  of  the  intended  user  community,  and  they  can 
present  evidence  to  support  their  claims  about  stylistic  and  language  design 
issues. 

In  1971)  Gerald  M.  Weinberg  wrote  The  Psychology  of  Computer  Programming  [2] 
to  trigger  the  study  of  computer  programming  as  a  human  activity.   Although 
most  of  the  book  concentrates  on  social  and  individual  activity  in  programming, 
Weinberg  also  hoped  to  encourage  language  designers  to  include  the  psychology 
of  computer  programming  as  a  new  dimension  in  their  design  philosophy. 

Since  the  appearance  of  Weinberg's  book,  an  increased  number  of  researchers 
have  conducted  psychological  experiments  on  programming  language  features. 
Using  inexperienced  programmers,  three  psychologists  concluded  that  nested  if 
programs  are  easier  to  understand  than  corresponding  goto  programs  [3].   Larry 
Weisman  listed  and  catagorized  factors  affecting  the  complexity  of  programs, 
began  development  of  a  methodology  for  studying  the  psychological  complexity 
of  computer  programs,  and  conducted  some  experiments  attempting  to  determine 
which  factors  or  combinations  of  factors  reduce  complexity  [U,5].   At  the 


University  of  Indiana,  memorization  tests  were  used  to  measure  the  effect  of 
program  structure  on  understanding  [6].   John  D.  Gannon,  in  his  Ph.D.  thesis, 
gathered  empriical  evidence  to  support  or  discredit  specific  language  design 
decisions  [7l-   Recently,  the  author  reported  the  results  of  an  experiment 
which  investigated  the  psychological  soundness  of  a  new  control  construct 
called  the  KAIL  selector  [8]. 

This  report  describes  yet  another  experiment  intended  to  experimentally 
verify  hypotheses  about  programming  language  design  issues.   It  explores 
alternatives   for  expressing  sequencing  constructs  in  computer-aided 
instruction  programs. 

1.2  The  KAIL  Project 

This  work  is  part  of  the  Automated  Computer  Science  Education  System 

(ACSES)  project  [9]  which  uses  the  PLATO  CAI  system  [10]  to  teach  elementary  computer 

science.   At  this  writing,  PLATO  runs  on  a  CDC  Cyber  73  dual  processor  with 

two  million  60-bit  words  of  extended  core  storage.   Attached  to  the  processor 

are  over  nine  hundred  plasma  display  terminals,  which  were  invented  by  the 

PLATO  group  for  this  project.   The  peak  comfortable  system  load  is  about 

four  hundred  simultaneous  users. 

Under  PLATO,  all  programs  are  written  in  a  special  system  language 
called  TUTOR  [ll]  which  mixes  FORTRAN-like  flow  of  control  with  peculiar 
CAI  sequencing  and  sophisticated  answer  judging  facilities.   The  language 
KAIL  is  an  attempt  to  influence  the  development  of  TUTOR  by  showing  that 
modern  control  constructs  and  sequencing  features  can  be  combined  with  the 
higher  level  features  of  TUTOR. 

1.3  General  Complaints  about  TUTOR 

Peculiar  sequencing  rules  (see  section  2.2)  make  TUTOR  code  difficult 
to  read;  moreover,  they  are  often  a  source  of  bugs.   In  order  to  understand 


a  listing,  an  author  must  memorize  the  rules  of  unit ,  jump,  help,  inhibit ,  etc. 
Bugs  arise  because  authors  forget,  for  instance,  that  goto  automatically  returns 
to  the  end  of  the  current  main  unit.   Often,  students  become  lost  in  lessons 
either  because  unanticipated  function  keys  are  active  or  normal  PLATO  defaults 
are  inactive.   These  errors  can  generally  be  attributed  to  an  author's  mis- 
understanding of  sequencing  rules. 

In  opposition  to  TUTOR,  a  CAI  language  which  follows  more  generally 
accepted  sequencing  rules  should  encourage  authors  to  write  more  reliable  code. 
The  experiment  described  in  this  report  tests  this  general  hypothesis. 


2.   THE  LANGUAGES  AND  HYPOTHESES 

2.1  The  Two  Languages  of  the  Experiment 

The  languages  designed  for  the  experiment  emphasize  differences  in 
sequencing  constructs  and  minimize  other  differences  as  much  as  possible. 
The  common  features  include  the  KAIL  selector  [12,  13]  which  was  previously 
tested  for  psychological  soundness  [8],  input-output  statements  similar  to 
those  found  in  TUTOR,  and  declarations  and  expressions  typical  of  high-level 
programming  languages.   One  language,  T,  incorporates  TUTOR  sequencing, 
and  the  other  language,  K,  contains  an  alternate  specification  for  sequencing, 
similar  to  that  proposed  for  KAIL.   The  important  differences  "between  K  and 
T  are  discussed  in  the  sections  below,  and  the  pertinent  pages  of  the 
reference  manuals  are  contained  in  appendix  A. 

2.2  TUTOR  and  the  T  Language 

TUTOR'S  approach  to  computer-aided  instruction,  in  its  simplest  form, 
is  based  on  a  sequence  of  "display-response"  frames.   A  TUTOR  program 
repeatedly  displays  information  on  the  screen  and  then  accepts  and  processes 
student  responses.   A  lesson  author  codes  a  particular  "display-response"  frame 
as  a  unit  and  strings  units  together  to  build  a  lesson  using  a  variety  of 
inter-unit  connections.   The  language  T  is  based  on  TUTOR. 
2.2.1  Program  Structure 

In  T,  a  unit  is  coded  as  a  procedure.   A  T  program  or  lesson  consists 
of  a  sequence  of  procedures  optionally  preceeded  by  global  variable  declarations 


<T-program>   : :=  lesson  <identifier> ; 

[<variable  declarations> ; ] 
<procedures> ; 
end 
<procedures>  :  :=  <procedure> | <procedure> ;  <procedures> 
<procedure>   : :=  <procedure  heading> ;  <statements> ;  end 
Procedure  nesting  and  local  declarations  are  not  permitted.   Execution  begins 
with  the  first  statement  in  the  first  procedure  and  flows  into  other  procedures 
via  internally  and  externally  initiated  transfers  of  control. 

The  system  initiates  an  internal  transfer  of  control  to  another  procedure 
when  it  encounters  one  of  the  branch  statements,  jump,  goto,  or  procedure  call. 
<branch  statements>   : :=  jump  <procedure  name> 

J  goto  <procedure  name> 
| <procedure  name> 
If  control  enters  a  procedure  via  jump,  entry  into  a  lesson,  or  an  externally 
initiated  transfer  of  control,  it  is  a  main  procedure;  and  if  control  enters 
a  procedure  via  goto  or  procedure  call,  it  is  an  attached  procedure.   Attached 
procedures  always  return  control  to  a  procedure  in  the  calling  routine.   For 
procedure  calls,  control  returns  to  the  statement  textually  following  the  call; 
for  goto ,  control  returns  to  the  end  of  the  main  procedure. 
2.2.2  Exception  Handling 

Externally  initiated  transfers  of  control  are  called  exceptions.   A 
student  presses  a  function  key  to  raise  an  exception. 

<function  key>        : :=  next | back | help | lab | data 

|<shifted  function  key> 
<shifted  function  key>  : :=  nextl|backl|helpl| labl| datal 
To  process  exceptions,  the  system  maintains  an  internal  pointer  for  each 
function  key.   This  pointer  is  active  or  enabled,  if  the  pointer  references 


a  specific  procedure  in  the  lesson;  it  is  inactive  or  disabled  if  the  pointer 
is  null.   The  system  transfers  control  to  the  designated  procedure  when  a 
student  raises  an  exception  that  has  an  enabled  pointer. 

When  control  enters  a  main  procedure,  all  pointers  are  set  to  null 
except  the  next  pointer  which  is  set  by  default  to  the  textually  next 
procedure.   When  execution  reaches  the  end  of  a  main  procedure,  the  system 
automatically  pauses  to  wait  for  student  input;  and  if  the  student  presses 
next ,  control  passes  to  the  procedure  designated  by  the  next  pointer.   By 
executing  an  exception  enabling  statement,  a  lesson  author  can  explicitly  set 
any  pointer,  including  the  next  pointer. 

<exception  enabling  statement> : :=  <function  key  name>  <procedure  name> 
For  example , 

help  hint ; 
helpl  big-hint; 
causes  the  help  pointer  to  reference  procedure  hint  and  the  helpl  pointer  to 
reference  procedure  big-hint. 

The  additional  semantics  of  exception  handling  in  T  depend  on  the  function 
key  name  in  an  exception  enabling  statement.   Next ,  nextl,  back,  and  backl 
evoke  lesson  sequences,  and  help,  helpl ,  lab,  labl ,  data,  and  datal  evoke 
help  sequenceE.   Lesson  sequences  are  completely  independent  of  any  execution 
history;  help  sequences  are  not. 

When  a  student  raises  an  exception  which  evokes  a  help  sequence,  the 
system  designates  the  current  main  procedure  as  the  base  procedure  and  sets 
an  internal  pointer  to  it.   Following  the  help  sequence,  control  returns  to 
the  beginning  of  the  base  procedure.   As  shown  in  figure  1,  a  help  sequence 
consists  of  one  or  more  main  procedures.   In  a  help  sequence,  the  system  not 
only  enables  the  next  pointer  as  usual,  it  enables  the  back  and  backl  pointers 
as  well  and  associates  them  with  the  base  procedure.   When  execution  encounters 
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Figure  1.   Lesson  and  Help  Sequences  in  T.   If  the  help  sequence 
is  evoked  from  one  of  the  main  or  attached  procedures 
in  the  lesson  sequence,  the  main  procedure  in  the 
calling  routine  "becomes  the  base  procedure  to  which 
control   returns  following  the  help  sequence. 
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an  endhelp  in  place  of  a  simple  end,  the  next  pointer  is  set  to  the  base 
procedure,  and  the  help  sequence  terminates. 
2.2.3  Modifications 

T  modifications  statements  are  defined  as 

Modification  statement>  : :=  rewrite | echo | erasure | si ze<expression> 

I  rotate  <expression> |  long  <expression> 
These  commands  modify  input-output  statements.   For  the  -write  statement, 
rewrite ,  echo ,  and  erasure  set  the  mode,  and  size   and  rotate  designate  the 
character  size  and  angle  of  writing  respectively.   The  long  statement  limits 
the  length  of  an  input  string.   Modifications  are  set  and  reset  dynamically, 
and  their  effect  extends  beyond  procedure  boundaries. 
2.2.U  Hidden  Side  Effects 

When  the  execution  of  a  statement  causes  an  action  unrelated  to  the 
main  function  of  the  statement  or  not  denoted  by  the  statement  name,  this 
action  is  a  hidden  side  effect.   For  example  the  T  jump  statement  causes  a 
screen  erase.   Hidden  side  effects  permeate  the  T  sequencing  features.   For 
each  main  procedure,  the  system  automatically  generates  a  screen  erase  at 
the  beginning  and  a  pause  at  the  end.   If  a  lesson  author  wishes  to  prevent 
these  defaults,  an  inhibit  erase  statement  nullifies  automatic  screen  erase; 
and  a  jump  statement  transfers  control  without  pausing. 

In  lesson  sequences,  the  system  automatically  associates  next  with  the 
textually  next  procedure.   In  help  sequences,  it  automatically  associates 
back      backl  with  the  base  procedure,  and  activates  next  as  in  lesson 
sequences  except  on  encountering  endhelp. 

Other  features  such  as  the  returning  goto  can  also  be  considered  as 
hidden  side  effects. 


2. 3  Hypotheses 

There  are  at  least  four  problems  with  T  sequencing  constructs.   Each 
has  led  to  a  hypothesis  tested  in  this  experiment. 
2.3-1  Uniformity 

T  lacks  uniformity.   One  example  is  the  nonuniform  "behavior  of  procedures. 
Syntactically,  procedures  appear  identical,  but  the  semantics  depend  on  whether 
a  procedure  connects  into  a  sequence  via  jump,  goto,   subroutine  call,  textual 
sequential  ordering,  or  student  raised  exceptions.   A  common  error  is  to 
insert  an  attached  procedure  between  two  main  procedures  and  then  evoke  it 
by  falling  through  from  the  textually  preceeding  main  procedure.   When  languages 
lack  uniformity,  many  rules  and  exceptions  to  rules  are  needed.   This  causes 
infrequently  used  constructs  to  be  forgotten,  makes  deviation  from  the  basic 
norm  difficult,  and  discourages  casual  users  from  gaining  in  depth  under- 
standing.  UNIFORMITY  HYPOTHESIS:   programs  would  be  easier  to  understand 
and  use  if  written  in  a  language  where  syntactically  similar  constructs  have 
similar  semantics. 
2.3.2  Orthogonality 

T  lacks  orthogonality.   When  language  facilities  are  highly  independent, 
they  are  said  to  be  orthogonal.   An  example  of  interdependence  in  T  is  the 
tight  binding  of  the  semantics  of  lesson  and  help  sequences  to  particular 
function  keys.   This  binding  discourages  authors  from  using  exceptions  in  other 
contexts  where  they  might  be  useful.   For  example,  in  T  it  is  difficult  to 
code  an  on-page  help  sequence,  a  help  sequence  where  help  text  appears  on  the 
same  page  as  instructional  text  and  where  control  resumes  at  the  point  of 
interruption  rather  than  the  beginning  of  the  base  procedure.   One  solution 
to  this  problem  might  be  to  introduce  a  special  exception  handling  statement  with 
its  own  rules  and  semantics,  but  this  only  compounds  the  problem.   Features 
which  are  interdependent  and  tightly  bound  together  lack  generality  and 
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thus,  create  a  need  for  further  extensions,  rules,  and  exceptions  to  rules. 
ORTHOGONALITY  HYPOTHESIS:   an  orthogonal  language  would  be  easier  to  under- 
stand, modify,  and  use. 
2.3-3  Dynamic  Execution 

T  depends  extensively  on  dynamic  execution.   T  exception  enabling 
statements  are  activated  dynamically.   This  allows  an  author,  for  instance, 
to  enable  the  help  key  in  an  attached  procedure  far  removed  from  the  calling 
routine  where  the  exception  may  be  raised.   It  also  allows  an  author  to 
sprinkle  several  specifications  for  a  single  function  key  throughout  a 
procedure  and  activate  them  depending  on  the  flow  of  control.   T 
modifications  also  depend  on  dynamic  execution.   A  common  mistake  is  to  set 
size  and  then  forget  to  reset  it.   Dyanmic  execution  features  are  error- 
prone  because  they  impose  fewer  restrictions  on  the  author.   DYNAMIC  EXECUTION 
HYPOTHESIS:  by  properly  limiting  the  dynamic  nature  of  these  and  similar 
statements,  it  is  likely  that  the  resulting  language  would  be  less  error-prone. 
2.3.1*  Hidden  Side  Effects 

T  sequencing  constructs  have  many  hidden  side  effects.   Each  of  these  is 
useful  and  makes  coding  easier  provided  an  author  understands  them  all  and 
uses  them  correctly.   T  authors  commonly  forget  to  inhibit  unwanted  side 
effects  and  often  use  explicit  statements  where  basic  defaults  are  already 
provided.   Also,  different  needs  may  create  situations  which  oppose  basic 
defaults.   These  needs  usually  force  authors  to  write  awkward  sections  of  code. 
Sometimes  further  extensions  (e.g.  on-page  help)  might  alleviate  these  awkward 
situations,  but  the  extensions  often  impose  new  defaults  or  additional  side 
effects.   The  number  of  rules  and  exceptions  to  rules  soon  grow  beyond  an 
author's  ability  to  remember.   HIDDEN  SIDE  EFFECTS  HYPOTHESIS:   although 
hidden  side  effects  usually  reduce  the  amount  of  coding,  it  is  likely  that 
by  eliminating  most  of  them,  a  language  would  be  easier  to  understand  and  use. 
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2.U     The  K  Language 

K  was  designed  to  propose  alternate  solutions  to  T's  sequencing  constructs 
It  stresses  uniformity,  orthogonality,  and  static  definition  and  minimizes  the 
use  of  hidden  side  effects. 
2.U.1  Program  Structure 

A  K  program  is  defined  as 

<K-program>  : :=  lesson  <identifier> ;  <constructs>;  end 

<constructs> : :=  <construct> | <construct> ;  <constructs> 

<construct>  : :=  <statement> 

| <variable  declaration> 
I <procedure  declaration> 
|  <exception  declaration 
Procedures  contain  local  variable  and  exception  declarations  "but  not  local 
procedure  declarations.   Execution  begins  with  the  first  statement  and  ends 
with  the  last  statement.   Only  explicit  calls  evoke  procedures;  thus,  K 
treats  all  procedures  uniformly. 
2.U.2  Exception  Handling 

In  K,  exception  handling  is  defined  as 

<exception  declaration>  : :=  on  <function  key>  [ <exception  handler> ] 
<exception  handler>      : :=  <statement> 
K  enables  exceptions  statically  and  thus,  avoids  dynamic  execution  as  a 
means  of  activation.   If  an  exception  is  raised  in  a  block  containing  an 
associated  exception  declaration,  control  immediately  passes  to  its  handler. 
After  executing  the  handler,  control  continues  with  the  statement  textually 
following  the  handler  unless  a  goto  explicitly  transfers  control  elsewhere. 
The  principle  of  orthogonality  is  observed  because  the  function  key  name  in  an 
exception  declaration  and  the  handler  are  totally  independent. 
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To  keep  an  exception  active  in  a  called  procedure,  the  exception  must 
be  passed  as  a  parameter.   A  parameter  list  in  K  is  defined  as 

<parameter  list>  :  :=  <parameter  set> | <parameter  set>  ;  <parameter  list> 

<parameter  set>   ::=  <type> [  result  ]  <identifier  list> 

| on  <function  key  list> 

<type>  ::=  integer | real | string  ( <integer> ) 

When  the  system  detects  an  exception  in  a  procedure  associated  with  an 
exception  declaration  in  its  parameter  list,  control  immediately  returns,  the 
system  raises  the  exception  in  the  calling  routine. 
2.U.3  Modifications 

K  modifications  are  defined  as 

<modifications>      :  :=  { Modification  list>} 

Modification  list>  :  :=  <modification>  |  <modification>  ,  <modifications> 

<modification>      ::=  rewrite | echo | erasure | size  <expression> 

| rotate  <expression> |  long  <expression> 
K  modifications  avoid  dynamic  execution  problems  because  a  modification  only 
affects  the  statement  to  which  it  is  attached.   Thus, 
{size  0.75)  [  at  ^29;   {size  3)write  TITLE; 
at  605 ;  write  sentence;] 
writes  "TITLE"  in  size  3,  but  writes  "sentence"  in  size  0.75*   When  a  state- 
ment to  which  modifications  are  attached  includes  a  procedure  call,  the 
modifications  carry  into  the  procedure.   For  example,  the  following  code 
places  a  box  on  the  screen  and  then  erases  it. 
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drawbox(x,y ) ;   { erasure } drawbox ( x,y ) ; 


procedure  drawbox( integer  x,y) ; 

draw  x,y. .x+100,y. .x+100,y+100..x,y+100. .x,y; 
end; 
2.U.U  Hidden  Side  Effects 

K  contains  no  hidden  side  effects.   The  language  requires  every  action 
to  be  explicit. 


3.   THE  EXPERIMENT 

3.1  Overview 

To  test  the  hypotheses  stated  in  chapter  2,  it  was  necessary  to  devise 
a  method  to  measure  a  subject's  ability  to  understand  and  use  a  language. 
Several  factors  were  involved. 

Personal  factors : 

1.  Educational  experience 

2.  Knowledge  of  programming  language 

3.  Basic  intelligence 
h.      Motivation 

Observable  factors : 

1.  Accuracy  in  coding 

2.  Speed  at  finding  and  correcting  program  bugs 

3.  Facility  at  finding  and  correcting  bugs  and  at  making 
modifications 

Together,  these  factors  formed  a  model  of  subject  understanding.   This  model 
of  a  subject's  ability  to  perform  programming  tasks  broke  the  hypotheses  down 
into  tests  involving  measureable  variables. 

For  the  experiment,  subjects  were  divided  into  two  groups,  the  T 
language  group  and  the  K  language  group.   The  effects  of  personal  factors  were 
controlled  for  by  grouping  subjects  so  that  background  and  motivation  in 
both  groups  were  similar.   After  receiving  instruction  in  their  assigned 
languages,  subjects  were  asked  to  debug  and  modify  a  reasonably  large  lesson 
consisting  of  about  500  lines  of  code.   Subjects  performed  this  task  on-line 
while  the  system  gathered  data. 
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3.2  Subjects 

The  experiment  took  place  during  the  Fall  Semester,  1975 »  and  the 
subjects  were  students  in  CS  317.   Professor  W.  Hansen,  instructor,  required 
each  student  to  participate  in  the  experiment.   As  part  of  the  course,  the 
students  learned  TUTOR  and  used  it  to  code  a  substantial  CAI  lesson.   In 
order  to  expose  them  to  the  additional  language  concepts  needed  for  the 
experiment,  the  course  included  a  section  on  KAIL.   Each  student  read  through 
lesson  KAIDS  [13],  an  introduction  to  KAIL,  coded  the  control  flow  for  a 
typical  KAIL  lesson,  and  took  a  quiz  to  test  their  comprehension. 

Based  on  this  quiz  and  a  background  information  sheet,  subjects  were 
divided  into  two  groups.   To  accomplish  this  division,  each  subject  received 
a  score  in  the  indicated  range  on  the  following  items : 
quiz  score  (0  -  37), 

status  at  the  university  (FR=1, .  .  .  ,C-1=5 ,  G2=6) 
number  of  CS  courses  taken  (0  -  6), 
understanding  of 

ALGOL  68  (1  -  5) 
KAIL  (1-5), 
TUTOR  (1  -  5 ) , 
PLATO  experience  (l  -  10). 
Subjects  were  ranked  according  to  the  sum  of  these  scores.   (Those  with 
identical  sums  were  ranked  alphabetically.)   Using  this  ranking,  subjects 
received  a  group  assignment  according  to  a  predetermined  random  grouping 
schema.   The  quiz,  background  information  sheet,  and  subject  grouping  schema 
are  contained  in  appendix  B. 

3. 3  Lectures  on  T  and  K 

Once  divided  into  groups,  subjects  learned  their  assigned  language  in  a 
one-hour  lecture  given  by  the  author.   Since  they  were  already  familiar  with 
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TUTOR  and  KAIL,  it  was  only  necessary  to  review  the  applicable  language 
features  for  each  group  and  emphasize  important  points.   Because  of 
scheduling  problems ,  subjects  attended  these  lectures  in  small  subgroups 
ranging  in  size  from  one  to  six.   Neither  language  group,  however,  gained 
an  advantage  by  having  more  small  subgroups.   At  these  lectures  subjects 
signed  up  for  their  session  on  PLATO;  these  were  scheduled  between  5  and  8 
days,  after  their  lecture. 
3.U  The  PLATO  Session 

The  on-line  portion  of  the  experiment  proceeded  as  depicted  in  Figure  2. 
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Figure  2.   Overview  of  the  Experiment 
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Each  subject  was  asked  to  debug  and  modify  lesson  "intrep"  on  integer 
representations  written  in  the  subject's  assigned  language,  K  or  T.   The 
logical  structure  for  "intrep"  was  as  depicted  in  Figure  3.   The  solid  arrows 
indicated  paths  through  the  lesson;  the  broken  arrows  indicated  paths  to 
the  quiz.   A  student  taking  "intrep"  was  to  complete  every  section  before 
attempting  the  quiz  (i.e.,  before  traversing  a  broken  path). 
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Figure  3.   Flow  Diagram  for  "intrep".   Solid  lines  indicate  paths 
through  the  lesson;  broken  lines  indicate  paths  to  the  quiz. 

Several  on-line  aids  were  available  during  the  experiment: 

(1)  A  complete  program  listing.   The  listings  of  "intrep"  for  the 
the  two  languages  appear  in  Appendix  C. 

(2)  An  editor  with  the  following  commands: 
Fn  to  move  forward  n  lines 

Bn  to  move  backward  n  lines 
In  to  insert  texx  after  line  n 
Rn  to  replace  line  n 
Dn  to  delete  n  lines 
Xt  to  locate  text  t 

(3)  A  reference  manual.   The  essential  pages  of  the  reference 
manuals  for  K  and  T  are  in  Appendix  A.   These  contained  syntax 
diagrams  and  a  minimal  explanation  of  the  semantics. 
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{h)      Program  execution.   A  subject  could  execute  "intrep"  to 

assist  in  locating  and  fixing  program  bugs. 
(5)   The  flow  diagram  for  "intrep",  figure  3. 

In  the  debugging  section,  subjects  had  25  minutes  to  find  as  many  of 
the  bugs  as  possible.   The  system  seeded  the  program  with  bugs  one  at  a  time 
and  in  a  specific  order.   Thus,  at  any  given  moment,  the  program  contained 
only  one  bug.   Their  task  was  to  find  it  and  fix  it. 

All  bugs  were  logic  errors  and  typified  those  which  might  actually  arise 
in  programming  "intrep".   Naturally,  bugs  were  selected  which  related  to 
the  hypotheses,  but  care  was  taken  to  keep  them  typical  and  thus,  hopefully, 
fair. 

To  help  subjects  identify  a  bug,  the  system  directed  them  through  a 
path  in  the  execution  of  the  lesson  leading  to  the  problem  and  then  gave  a 
one  line  explanation.   To  shorten  the  path  leading  to  an  error,  the  system 
began  each  path  at  a  convenient  point,  not  necessarily  the  title  page.   Sub- 
jects were  able  to  execute  "intrep"  as  desired  to  help  locate  the  bug. 

When  a  subject  pressed  the  term  key  and  entered  "editor"  the  program 
listing  became  available  for  editing.   While  editing,  subjects  could  return 
again  to  the  execution  of  "intrep"  with  the  bug.   Any  changes  to  the  code, 
however,  were  not  reflected  in  the  execution  of  the  program. 

After  subjects  fixed  a  bug  (or  gave  up),  the  system  presented  a  solution 
to  the  problem.   The  subject's  solution  was  graded  later  by  hand. 

As  an  example,  figures  kt    55  and  6  show  one  bug  and  its  solution,  first 
in  K  and  then  in  T. 
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After  completing  the  debugging  section,  subjects  were  given  an  additional 
25  minutes  to  modify  a  bug-free  listing  of  "intrep"  according  to  the  following 
specifications : 
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While  editing,  subjects  could  refer  to  these  specifications  as  often 
as  desired.   The  system  imposed  no  constraints  on  how  subjects  were  to 
accomplish  the  task;  thus,  they  could  modify  the  program  as  they  wished. 
Their  efforts  were  graded  later  by  hand. 
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k .   RESULTS 

k. 1  Measures  of  Ability 

As  indicated  in  section  3-1,  a  subject's  ability  to  comprehend  and  per- 
form was  measured  in  several  ways: 

(1)  accuracy  in  coding: 

(a)  subjective  scores 

(b)  number  of  coding  errors 

(c)  type  of  coding  errors 

(2)  speed  at  finding  and  correcting  each  program  bug: 

(a)  time  spent  executing  "intrep" 

(b)  time  spent  searching  the  listing 

(c)  time  spent  altering  the  listing 

(d)  time  spent  using  the  aids 

(e)  total  time 

(3)  facility  at  finding  and  correcting  each  bug  and  at  making 
modifications : 

(a)  number  of  times  "intrep"  was  executed  for  the  debugging 
section,  and  number  of  times  the  specifications  were 
accessed  for  the  modifying  section 

(b)  number  of  times  editor  searching  commands  were  used 

(c)  number  of  changes  of  the  listing 

(d)  number  of  times  aids  were  referenced 

(e)  total  number  of  editing  keypresses 

These  performance  statistics  appear  in  appendix  D,  E,  and  F,  and  the  important 
results  are  discussed  below. 
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k.2     Difficulties 

Several  difficulties  were  encountered  during  the  experiment. 

Two  K  subjects  and  one  T  subject  dropped  out  of  the  experiment.   This 
left  an  imbalance  of  13  K  subjects  and  15T  subjects.   T-tests  showed,  however, 
that  this  did  not  bias  the  experiment. 

During  one  session,  the  system  crashed.   Because  this  problem  was 
anticipated,  experiment  sessions  on  PLATO  were  limited  to  at  most  four  sub- 
jects, two  from  each  group.   The  crash  caused  the  loss  of  partial  data  from 
two  T  language  subjects  and  one  K  language  subject. 

In  other  sessions,  three  subjects  accidentally  terminated  the  experiment 
prematurely.   A  T  language  subject  accidently  hit  stopl ,  a  built-in  escape 
feature  from  any  PLATO  lesson.   Misunderstanding  the  directions  in  the  modifica- 
tion section  caused  two  K  language  subjects  to  prematurely  press  the  combina- 
tion of  keys  provided  for  exiting  the  experiment.   In  all  of  these  cases, 
partial  data  was  lost. 

In  analyzing  the  results,  an  unexpected  number  of  language  interference 
errors  appeared.   K  subjects  used  a  few  TUTOR-like  constructs,  and  T  subjects 
used  a  few  KAIL-like  constructs.   Despite  the  interference,  it  was  relatively 
easy  to  decide  what  caused  a  subject  to  choose  a  particular  syntactic  form 
or  expect  some  specific  semantic  action. 
k. 3  Background 

As  discussed  in  section  3.2  subjects  were  grouped  so  that  background 
and  ability  would  be  equal.   An  analysis  of  the  statistics  confirmed  this; 
there  were  a  few  specific  areas,  however,  in  which  unexpected  differences 
occurred.   A  typical  subject  in  the  K  language  group  was  a  first  year 
graduate  student,  while  a  typical  subject  in  the  T  language  group  was  nearly, 
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but  not  quite,  a  senior  at  the  university.   This  was  almost  close  enough  to  b 

be  considered  significant  (p  ^_   .059).   More  K  subjects  had  taken  CS  323, 

system  programming  (p  <_  .02U);  and  K  subjects  better  understood  ALGOL-W 

(p  _;_  . OlU).   Both  groups  had  similar  PLATO  experience  and  KAIL  quiz  scores. 

Table  1  summarizes  the  background  and  ability  data. 
Personal  Data 

K 

university  status  U.38 

course  taken  3.38 

language  understanding  27-38 

PLATO  experience  U.31 

quiz  score  28.92 

Table  1.   Summary  of  Background  and  Ability  Data 

k.h     Debugging  Statistics 

The  debugging  section  of  the  experiment  was  completed  by  12  of  13  K 
subjects  and  12  of  15  T  subjects.   The  other  k   encountered  the  difficulties 
discussed  in  section  U.2.   Table  2  shows  the  extent  to  which  subjects  from 
each  group  worked  the  problems.   T  subjects  worked  more  problems  (5.8  on  the 
average)  than  K  subjects  (U.8  on  the  average),  but  this  is  not  statistically 
significant.   Bugs  1  through  5  are  discussed  below;  not  enough  subjects 
worked  problems  6  through  10  to  allow  statistical  analysis. 
U.U.I  Bug  1 

The  first  bug  tested  the  dynamic  execution  hypothesis.   A  title  in  "intrep" 

was  supposed  to  be  written  in  size  l-5»  "but  the  statement 

write  C.  FIBONACCI  REPRESENTATION, 
wrote  it  in  size  0.   K  language  subjects  should  have  fixed  this  bug  with 

{size  1.5}  write  C.  FIBONACCI  REPRESENTATION; 
and  T  language  subjects  should  have  fixed  it  with 

size  1.5;  write  C.  FIBONACCI  REPRESENTATION;   size  0; 
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Number 

Problem 

Problem 

at 

Problem 

1 

K 

11 

0 

2 

T 

Ik 

0 

1 

2 

K 

13 
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T 

lk 

0 

0 

3 

K 

10 

2 

0 

T 

12 

0 

2 

k 
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8 

2 

0 

T 

11 

2 

0 

5 

K 

5 

0 

3 

T 

8 

0 

3 

6 

K 

2 

2 

0 

T 

5 

2 

0 

7 

K 

1 

0 

0 

T 

3 

0 

0 

8 

K 

0 

1 

0 

T 

0 

2 

0 

9 

K 

0 

0 

0 

T 

0 

0 

0 

10 

K 

0 

0 

0 

T 

0 

0 

0 

Table  2.   The  Extent  to  which  Subjects  Worked 
the  Problems.   There  were  13  K  subjects  and  15 
T  subjects. 

Only  1  of  ik   T  subjects  who  attempted  the  problem  solved  it  correctly, 
whereas  5  of  11  K  subjects  solved  it  correctly  (table  3).   An  exact  [lk]    showed 
this  was  significant  (p  <_  .039)*   Moreover,  only  3  subjects  in  the  T  group 
made  any  attempt  to  reset  size,  and  another  3  had  the  failure  to  reset  size 
as  their  only  error.   The  11  K  subjects  made  8  errors;  the  lk   T  subjects 
made  18  errors.   Timing  and  editing  data  showed  essentially  no  differences. 
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K 


number  of  errors 
0      1      >  2 


5      k  2 

1     9      !+ 


Table  3.   Error  Contigency  Table  for  Bug  1 

This  data  supports  the  dynamic  execution  hypothesis.   Subjects  committed 
fewer  coding  errors  when  using  static  modifications. 
k.k.2     Bug  2 

The  second  bug  tested  the  hidden  side  effects  hypothesis.   In  the  K 
listing,  an  unwanted  erase  was  placed  at  the  end  of  a  help  sequence;  in  the 
T  listing,  a  necessary  inhibit  erase  was  deleted.   Of  13  K  subjects,  8 
performed  well;  of  lU  T  subjects,  8  also  performed  well.   Two  T  subjects, 
however,  placed  inhibit  erase  in  the  wrong  procedure;  apparently,  they  under- 
stood the  problem  but  not  the  rules  of  inhibit  erase.   The  K  group  introduced 
0.23  errors  per  person;  the  T  group  introduced  0.31  errors  per  person.   Although 
K  subjects  spent  less  time  altering  the  listing  (p  <_  .009),  this  only 
indicates  that  inserting  takes  more  time  than  deleting.   No  other  significant 
j  differences  emerged. 

Apparently,  it  was  no  more  difficult  to  detect  the  absence  of  an  inhibit 
erase  and  insert  it  than  to  find  and  delete  an  unwanted  erase.   This  bug  was 
probably  too  simple  to  reveal  any  significant  information  about  the  hidden 
side  effects  hypothesis. 
;  k.k.3      Bug  3 

The  third  bug  considered  program  structure  and  exception  handling  and  tested 
i  hypotheses  relating  to  uniformity  and  dynamic  execution.   Coding  CAI  lesson 
!  sequences  with  traditional  constructs  seemed  to  demand  a  level  of  indirection 
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(goto  label  1  ...  1:   call  procedure  p) ,  but  T  sequencing  (continue  at 
procedure  p)  did  not.   Figure  7  shows  K's  indirect  method  of  going  back  to 
the  previous  frame;  figure  8  shows  T's  direct  method.   The  bug  was  seeded 
into  the  programs  by  deleting 

on  back  goto  ttl; 
from  K's  listing  and 

back  title; 
from  T's  listing. 

Only  1  of  10  K  subjects  who  completed  the  problem  solved  it  correctly, 
whereas  7  of  12  T  subjects  solved  it  correctly  (table  h) .   An  exact  test 
[li+]  showed  this  was  significant  (p  <_   .026).   Of  the  10  K  subjects,  8 
entered  goto  <procedure  name> ;  moreover,  2  T  subjects  used  KAIL  constructs  but 
also  entered  goto  <procedure  name> .   The  10  K  subjects  made  l6  errors;  the  12 
T  subjects  made  only  9  errors.   On  the  average,  the  T  subjects  scored  8.1 
(10  possible),  and  the  K  subjects  scored  5.6  (p  <_  .013).   Moreover,  even  though 
the  amount  of  code  to  be  inserted  for  a  correct  solution  was  nearly  identical 
for  both  languages.   K  subjects  spent  more  time  inserting  (p  <_  .030),  perhaps 
because  they  were  unsure  of  themselves. 


K 


number  of  errors 
0      1      >  2 


1 

i 
3 

'      ■  " 

6 

7 

5 

0 

Table  h.      Error  Contingency  Table  for  Bug  3 
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Contrary  to  the  general  hypothesis  (section  1.3),  traditional  sequencing 
in  K  resulted  in  poorer  performance.   The  K  language  subjects  seemed  confused 
by  going  one  place  to  get  somewhere  else. 


ttl.      title; 
indx.      index; 


procedure  title; 


end  title; 
procedure  index; 

on  back  goto  ttl: 


end  index; 

Figure  ?•   Handling  back  in  K 
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procedure  title; 


end  title; 


procedure  index; 
back  title; 


end  index; 

Figure  8.   Handling  back  in  T 

On  the  other  hand,  2  T  subjects  placed  "back  title"  in  the  wrong  procedure. 
In  both  instances,  these  subjects  may  have  expected  back  to  remain  active 
once  it  was  enabled  since  under  this  assumption,  they  essentially  solved  the  problei 
correctly. 

This  lends  support  to  both  the  uniformity  and  dynamic  execution  hypothesis 
The  2  T  subjects  seemed  to  be  confused  about  the  interaction  between  dynamic 
activation  of  exceptions  and  entrance  into  a  main  procedure. 
k.k.k     Bus  k 


ei 


The  fourth  bug  tested  the  hidden  side  effects  hypothesis.  In  T,  the  jump 
statement,  in  addition  to  directing  control  to  a  designated  procedure,  erased 
the  screen.  The  goto  only  directed  control  elsewhere  and  could  often  be  used 
in  place  of  a  jump.  Bug  k  was  a  screen  overwrite  problem.  An  erase  was  left 
out  of  the  K  listing;  a  goto,  instead  of  a  jump,  was  used  in  the  T  listing. 
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In  T,  10  of  11  subjects  gave  a  correct  solution,  but  only  1  replaced 
the  goto  with  a  jump.   The  rest,  9  of  10,  gave  an  alternate  solution:   they 
properly  inserted  erase  in  the  appropriate  procedure.   Clearly,  T  subjects 
found  it  easier  to  fix  the  bug  directly.   In  K,  5  of  8  solved  the  problem 
correctly.   Both  groups  fixed  the  bug  in  the  same  manner;  however,  K  subjects 
inserted  more  often  (p  <_  .0^5),  activated  more  searching  options  (p  ^_   .015), 
and  pressed  more  editing  keys  (p  <_  . 0U0).   There  is  no  clear  explanation 
why  K  subjects  edited  more  than  T  subjects. 

The  results  lend  support  to  the  hidden  side  effects  hypothesis  because  the 
explicit  solution  was  preferred.   Underlying  implicit  effects  were  more  difficult 
to  find. 
I.U.5   Bug  5 

The  fifth  bug  tested  the  hidden  side  effects  hypothesis.   Because  help 
sequences  in  T  returned  to  the  beginning  of  the  base  procedure,  procedures 
had  to  be  broken  at  unusual  junctures;  and  because  the  screen  erased 
automatically  when  entering  a  main  procedure,  normal  defaults  had  to  be 
overridden.   Solving  both  these  problems  at  once  resulted  in  particularly 
awkward  and  unusual  code.   Figure  9  shows  the  essential  statements  in  a  section 
of  T  code  where  unusual  coding  was  necessary.   The  inhibit  erase  prevented 
an  unwanted  screen  erase  on  entering  the  help  sequence,  mrhelp.   The  erase 
in  mixedr2  was  essential  because  the  automatic  erase  which  normally  would 
have  occurred  in  falling  through  from  mixedrl  was  inhibited.   Figure  10 
shows  the  corresponding  section  of  K  code.   In  both  instances,  the  bug 
was  created  by  removing  the  erase. 

Of  the  o  T  subjects  who  attempted  to  solve  this  problem,  3  removed 
inhibit  erase  from  procedure  mixedrl  which  fixed  the  immediate  problem  but 
introduced  another  bug.  All  5  K  subjects  who  attempted  this  problem  solved 
it  correctly.   K  subjects  scored  lU.O  of  15  in  a  subjective  measure  which 
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adjusted  the  score  to  give  more  credit  to  those  who  did  well  quickly,  but 
T  subjects  scored  only  9.6.   This  is  significant  (p  <_  . 0^9). 

This  supports  the  hidden  side  effects  hypothesis  and  shows  how  combinations 
of  hidden  side  effects  interfere  with  one  another  ot  produce  unexpected  bugs. 


procedure  mixedrad; 


end  mixedrad; 
procedure  mixedrl; 


inhibit  erase; 


help  mrhelp; 


at  3020;  write  Press  NEXT  to  continue. ; 
end  mixedrl; 
procedure  mixedr2; 
erase ; 


end  mixedr2: 


Figure  9-   Section  of  T  Code 
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procedure  mixedrad; 


on  help  mrhelp ; 


at  3020;   write  Press  NEXT  to  continue.; 
pause  next ; 
erase ; 


end  mixedrad; 
Figure  10.   Section  of  K  Code 

h. 5  Modifying  Statistics 

The  modifying  section  of  the  experiment  was  completed  "by  10  of  13  K 
subjects  and  13  of  15  T  subjects.   The  other  5  encountered  the  difficulties 
discussed  in  section  k.2.      Refer  to  section  3.^-  for  the  spcifications  used  in 
the  modifying  section  of  the  experiment. 

K  subjects  made  several  errors  related  to  the  hypotheses  of  the  experiment. 
Most  of  these  related  to  the  hidden  side  effects  hypothesis  and  were  probably 
a  result  of  TUTOR  language  interference.   On  11  occasions  of  a  possible  20, 
no  pause  occurred  after  help  messages.   Subjects  may  have  been  counting  on 
a  nonexistent  automatic  pause  at  the  end  of  a  procedure.   K  subjects  also  seemed 
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confused  about  erasing  after  help  messages:   in  5  places  in  connection  with  the 
first  specification,  an  unnecessary  erase  appeared;  and  in  k   places  in  connection 
with  the  second  specification,  the  message  was  left  on  the  screen.   Contrary 
to  the  general  hypothesis  (section  1.3),  K  subjects  continued  to  include  errors 
of  the  from 

goto  <procedure> 
A  total  of  10  of  these  errors  crept  into  the  listings.   This  further  supports 
the  results  found  in  bug  3  and  shows  that  this  kind  of  indirect  transfer  to  a 
frame  processor  is  probably  unnatural. 

T  subjects  also  made  several  errors  related  to  the  hypotheses  of  the 
experiment.   With  regard  to  the  uniformity  hypothesis,  T  subjects  positioned 
5  of  a  possible  26  attached  procedures  in  sequences  of  main  procedures  so  that 
the  attached  procedures  would  also  be  executed  as  main  procedures.   In  the 
second  specification,  no  one  solved  the  pause  problem  correctly.   With  the 
expection  of  those  who  did  even  worse,  everyone  included  a  double  pause 
in  approximately  the  form  shown  in  figure  11.   One  pause  occurred  explicitly; 
the  other  occurred  implicitly  at  the  end  of  the  help  sequence.   This  relates 
to  the  hidden  side  effects  hypothesis,  but  it  also  relates  to  the  orhtogonality 
hypothesis  because  the  implicit  pause  at  the  end  of  the  help  sequence  is  bound 
to  rather  than  separate  from  the  semantics  of  the  help  sequence.   Other 
errors  were  related  to  the  hidden  side  effects  hypothesis.   On  9  of  26 
occasions,  subjects  did  not  take  advantage  of  normal  defaults  and  referred 
unnecessarily  to  next ,  back,  and  backl.   T  subjects  omitted  endhelp  in  5  of  26 
help  sequences  and  forgot  to  include  inhibit  erase  in  6  of  13  instances  where 
it  was  required. 
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procedure  sorry 

inhibit  erase ; 
at  3010;  write  sorry; 
pause  next  ,back  ,"backl ; 
at  3010;   erase  5; 
endhelp  sorry; 

Figure  11.   Double  pause  Problem 

All  subjects  in  both  groups  coded,  the  control  for  the  second  problem 
improperly.   The  specification  called  for  control  to  resume  at  the  point  of 
interruption  in  the  quiz.   Most  subjects,  however,  wrote  code  which  simply 
restarted  the  quiz;  and  no  one  even  attempted  to  force  control  to  resume  at 
the  interruption  point.   The  problem  was  difficult  enough  that  subjects  were 
unable  or  unwilling  to  solve  it. 

In  general,  observations  on  the  modification  section  of  the  program 
support  the  hidden  side  effects  hypothesis.   In  25  minutes,  10  K  subjects  made 
made  20  hidden  side  effects  related  errors  and  13  T  subjects  made  36  such 
'errors.   The  K  errors  seemed  to  be  based  on  assumptions  learned  from  TUTOR. 
1+.6  Correlations 

Appendix  F  contains  the  correlation  data  for  the  major  factors  in  the 
i experiment.   Using  score  as  a  measure  of  performance,  table  5  shows  how  it 

correlated  with  other  factors.   The  number  of  years  of  university  training  had 

i 

no  correlation  with  performance.   Other  background  data  correlated  highly  with 

scores  from  the  K  language  group,  but  not  so  highly  with  scores  from  the  T 

language  group.   Programmers  with  greater  insight  and  ability  performed  better 


36 

in  K,  but  insight  and  ability  did  not  seem  to  matter  so  much  in  T.   Perhaps 
this  was  because  T,  being  like  TUTOR,  was  more  familiar  to  all  subjects, 
whereas  K,  contianing  less  familiar  concepts,  allowed  experienced  programmers 
to  perform  better  and  hindered  less  experienced  programmers  even  more. 
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5.   CONCLUSIONS 

5.1  Summary  of  Results 

Although  some  significant  and  interesting  results  emerged,  it  was  not 
possible  to  state  conclusively  that  the  entire  set  of  K  sequencing  constructs 
was  "better  than  the  set  of  T  sequencing  constructs.   Instead,  the  results 
were  mixed. 

The  experiment  did  show,  however,  that  direct,  explicit,  and  simple 
constructs  were  best.   Indirect  frame  sequencing  in  K,  implicit  constructs 
in  T,  and  the  lack  of  a  simple  resume  feature  in  both  K  and  T  seemed  to  cause 
most  of  the  errors.   In  both  bug  3  and  the  modifying  section,  indirect 
sequencing  patterns  confused  K  subjects  and  resulted  in  poorer  performance. 
This  was  the  only  significant  contradiction  to  the  general  hypothesis  that 
more  generally  accepted  sequencing  rules  would  encourage  authors  to  write 
more  reliable  code.   Misunderstanding  hidden  side  effects  and  interference 
with  TUTOR  hidden  side  effects  hindered  T  and  K  subjects  respectively. 
Both  groups  committed  several  coding  errors  which  seemed  to  be  based  on 
expectations  about  implicit  actions  such  as  automatic  pause  or  erase. 
Because  T  lacked  an  on-page  help  feature,  and  because  K  lacked  a  resume  feature, 
there  was  no  simple  solution  to  the  control  problem  for  the  second  specification 
in  the  modifying  section  of  the  experiment.   Because  of  this  difficulty,  subjects 
were  unwilling  or  unable  to  force  control  to  resume  at  the  point  of  interruption. 
5.1.1  Unfiormity 

The  data  from  the  experiment  gave  support  to  the  uniformity  hypothesis. 
In  the  modifying  section,  T  subjects  introduced  errors  by  improperly  inserting 
attached  procedures  into  the  listing.   K's  uniform  treatment  of  procedures 
prevented  K  subjects  from  making  this  mistake. 
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5.1.2  Orthogonality 

The  data  from  the  experiment  supported  the  orthogonality  hypothesis.   In 
the  modifying  section,  all  T  subjects  found  the  double  pause  problem 
particularly  difficult  to  avoid.   In  K,  procedure  and  pause  were  independent, 
so  the  problem  did  not  occur. 
5.1-3  Dynamic  Execution 

The  experiment  confirmed  one  instance  of  the  dynamic  execution  hypothesis, 
When  dealing  with  dynamic  modifications  in  bug  1,  T  subjects  introduced  more 
coding  errors  than  K  subjects.   Less  dynamic  K  modifications  were  easier  to 
understand  and  use. 
5.1.]4  Hidden  Side  Effects 

The  results  supported  the  hidden  side  effects  hypothesis.   In  bug  U, 
T  subjects  preferred  to  fix  a  screen  overwrite  problem  directly  by  inserting 
an  erase  rather  than  indirectly  by  changing  a  goto  to  a  jump  obtaining  erase 
as  a  side  effect.   In  bug  5,  the  inhibit  erase/erase  phenomenon  caused  mis- 
understandings for  some  T  subject. 

As  exhibited  by  errors  which  emerged  from  the  modifying  section  of  the 
experiment,  T  subjects  misunderstood  aspects  of  hidden  side  effects  and 
K  subjects  allowed  the  TUTOR  knowledge  to  interfere.   Seme  K  subjects  left 
out  necessary  pauses  and  seemed  confused  about  erasing  after  help  messages. 
Many  T  subjects  referred  unnecessarily  to  next ,  back ,  and  backl  and  forgot 
to  include  a  needed  inhibit  erase. 
5 . 2  Recommendations 

These  recommendations  for  CAI  language  features  resulted  from  the  data 
of  the  experiment . 

1.   Allow  direct  jumps  to  frame  processors.   Because  CAI  is  heavily 

based  on  meaningful  sequence  of  display-response  frames,  an  author 
language  should  allow  programmers  to  directly  and  simply  code 
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frame  processing  and  sequencing.   The  experiment  demonstrated 
that  this  deficiency  in  K  resulted  in  poor  performance.   The 
data  did  not  imply,  however,  that  all  subroutines  should  he 
coded  as  frame  processors  as  in  T. 

2.  Limit  dynamic  execution  of  modifications .   Although  more  tests 
would  be  necessary  to  show  that  the  syntactic  form 

{<modifications>}<statement> 
is  best,  the  results  indicate  that  this  semantic  restriction  should 
be  placed  on  modification. 

3.  Eliminate  most,  if  not  all,  hidden  side  effects.   As  expected, 
hidden  side  effects  adversely  affected  performance.   More  testing 
would  be  necessary,  however,  to  firmly  state  that  all  hidden  side 
effects  should  be  eliminated  or  that  a  specific  few  should  be 
retained. 

k.      Include  a  resume  feature  as  part  of  exception  handling.   As 

exemplified  in  the  modifying  section  of  the  experiment,  the  lack 
of  this  facility  would  discourage  authors  from  writing  code  to 
permit  a  student  to  obtain  help  and  then  resume  at  the  point 
of  interruption.   This  need  occurs  commonly  in  CAI. 
5. 3   Suggested  Improvements 

Although  performance  measures  on  accuracy  in  coding  and  observations 
on  the  specific  kinds  of  coding  errors  produced  a  reasonable  number  of 
interesting  results,  statistics  on  the  speed  and  facility  at  fixing  program 
bugs  and  on  the  facility  at  making  modifications  were  disappointing.   It 
was  hoped  that  the  time  spent  executing  "intrep",  searching  the  listing,  and 
the  total  time  spent  locating  and  fixing  a  bug  would  yield  information  about 
which  constructs  make  debugging  faster  and  thus  easier,  and  it  was 
hoped  that  the  total  number  of  editing  keypresses  would  indicate  which 


constructs  and  program  structure  make  a  programmer  work  harder  to  debug  and 
modify  a  lesson.   The  experiment  would  have  been  improved  if  subjects  could 
have  observed  the  result  of  their  changes  when  executing  "intrep".   This 
would  have  forced  them  to  reconsider  their  fixes  and  modifications  in  light 
of  whether  or  not  they  actually  solved  the  problem.   It  is  likely  that 
more  obscure  constructs  would  have  caused  subjects  to  spend  more  time 
and  do  more  work  to  solve  the  problems.   As  it  was,  however,  the  system 
accepted  any  solution  and  left  it  unchallenged. 

The  number  of  interference  errors  was  larger  than  expected,  and  more 
K  subjects  committed  interference  errors  than  T  subjects.   This  indicates 
that  the  experiment  would  have  been  improved  by  spending  more  time  teaching 
the  assigned  languages  and  by  insuring  a  greater  similarity  in  a  subject's 
TUTOR  and  KAIL  background. 

Three  other  changes  might  have  improved  the  experiment. 

1.  The  correlation  between  university  status  and  performance  indicates 
that  this  factor  should  not  have  been  considered  as  part  of  a 
subject's  background  data. 

2.  Although  25  minutes  was  about  the  right  amount  of  time  for  the 
modifying  section  of  the  experiment,  subjects  needed  more  than 
25  minutes  for  the  debugging  section. 

3.  The  choice  of  bugs  for  this  kind  of  experiment  is  always  very 
difficult.   The  experiment  might  have  been  imporved  by  a  different 
choice  of  bugs,  but  the  chosen  set  seems  to  have  been  reasonable. 
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APPENDIX  A 
PORTION f»  OF  THE  REFERENCE  MANUALS  FOR  T  AND  K 

Each  page  in  the  on-line  reference  manuals  used  during  the  experiment 
gave  a  syntax  diagram  for  a  particular  construct  and  some  explanation  of  the 
semantics.   A  "box,  indicated  that  more  of  the  diagram  appeared  on  another  page. 

The  pages  of  the  reference  manuals  selected  for  this  appendix  contain 
the  essential  differences  between  T  and  K. 
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A.l     Pages    from  the  T   Reference  Manual 


P.    PROOF mM 


/ — r\ 


lesson  — (.  1  .  id} — .;  -V-<v.  var  decl  ) —  ; 

— ^ — '     o 


^ 


(u.    pre:  de<:  1  }■ 


■  T 


end  -H^i.  id)-3-*-  ; 


Execution  begins  with  the  first  statement  in  the  first 
procedure  and  end;  after  execution  of  the  lasu  statement  in 
the  l?3t  procedure. 


Examp 1 e : 

lesson  5.5  'hi; 


procedure  hi ; 

at  IE.  10; 
write  Hi ' ; 
end  h 1 ; 


end  sayhi 


U.  PPOCEDLIRE  DECLmRRTIOU 


procedure  — '  i .  idy 


jf  end 


L 


■Ct .  s  t  a  t  erne  nt  > 


LJ 


endhe 1 p 


/ 


Procedures  behave  like  TUTOR  units. 


Use  endhelg;  to  terminate  a  help  sequence, 


Examp 1 e : 


procedure  wa  i t ; 

at  302  7;   write  Press  NEXT; 

pause  next; 

at  3027;   erase  ir»; 

end  uwi  1 1 ; 
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T.  STATEMENT 


r 


I 


^\i_^!/ 


-v^=Zii> 


Examples: 


branch  lbl; 

lbl .  1  *  i  +  1; 

size   2.5;       write   CONGPmTULnTIONS; 


size   0; 


2.    STT1T 


/s.    selector)- 


-*-  *  — (e.    expV 


Exarnples: 


i    <s=    i  ♦  1 

fHi)    *   FKi)  ,'denom 
V   *  '(1,2,3,4,5) 

term    "  i  n:iev  "  ,  "contents" 
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M.  MODIFICATION 

rewr  i  t  e  


Modifications  alter  the  normal  behavior  of  certain 
statements.  Once  set,  a  mod 1 f i cat i on  remains  in  effect 
until  it  is  explicitly  reset. 


Examp lei: 

size  2;  rotate  45;   write  hypotenuse; 

size  0;   rotate  Q ; 
rewrite;   write  replace  it!;   echo; 


B.  BRANCH 


branch 


-<H^> 


v   _/ ~-> 

goto  — C  i  .  idy- 

r. — ~~\ 

j ump  — (  i  .  id/- 


(T^)I 


C^Ye 


XI 


.  evpW) 


These  statements  behave  like  the  TUTOR  commands  to 
which  thev  correspond.   '< id^"  corresponds  to  "do   <id>" 


Examples: 


goto  index 
jump  axxl 
branch  label 
prc-c  1 0 


^  "index"  is  a  procedure  name  $ 
tf    "axvi"  is  a  procedure  name  ^ 
i    "label"  is  a  label  i 
fi    "proclfl"  is  a  procedure  name  £ 


hi 


0.  INHIBIT 


i  nh  i  b  1 1  -f   V,, 


event  V-  ,  — «^-  term  -4r-  ,  ~^>-  event  V-^ 


These  statements  inhibit  normal  behavior  until 
explicitly  altered  or  until  entrance  into  a  new  main 
procedure . 


Examples: 


inhibit  data 
inhibit  back 
inhibit  erase 


£  makes  the  data  kav  inactive  ^ 
£  makes  the  back  key  inactive  £ 
^  inhibits  automatic  erasures  £ 


These  statements  behave  1  ike  the  TUTOR  commands  to 
which  they  correspond. 


Examples: 


next  tcpic3  i  sets  the  next  pointer  to  topics  i 
back  topic  1  f  sets  the  back  pointer  to  topic:  t 
help  hint  i   establishes  a  help  sequence,  hint  i 
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A. 2     Pages    from  the   K  Reference  Manual 


P.    PROORrtM 


les;.:n   — ■'  1 .     idV 


Example: 

lesson  sayhi ; 

at     1010; 
write   Hi ' ; 

end   sayhi 


statement 


v.    var  decl 


/     \    \u.    proc  decl/ 


.    on  de-rl  >■ 


J 


T.    STflTETEMT 


C 


) 


-'.  m .    mod  1  f  i  oat  i  on= 


Al  /C 


vr. 


stmt)- 


Exan  jp  1  • 


goto    lbl 

lbl .     i    «    i ♦ l 

(  sire   2.5    } write  COr-IGPflTlJLrtTIONS 
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M.  MODIFICATIONS 


ech 


Modifications  alter  the  normal  behavior  of  certain 
statements  to  which  they  ar-z   attached.   This  statement  may 
include  a  procedure  call  in  uhich  case  the  modification  is 
in  effect  whi le  the  procedure  is  active. 

Examples: 

{  size  2;  rotate  -45  )write  hypotenuse 
(  rewrite;  } write  replace  it1 
{  disable  back, back 1  } lethargic 


U.  PROCEDURE  DECLARATION 


procedure 

. t.  statement  / 


<^>T7^: 


parameters) —  ) 


1 


v.  v*r  decT) j —  ;  -tt-  £2^.  ~%i  •  idr 


5.  on  derl 


Exampl- 


procedure  wa  i  t ; 

at  302  7;   writ-:  Press  NEXT; 

pause  ne-t; 

at  3027;   ersse  10; 

end  wa i t ; 
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\.    PriF.'MMETEIFS 


string    —    (   — in.     integer  J —  ) 


/  integer   N  / 

/ 

\  l 


\\  real 


1/ — x     I     1. 

result  -^\i.    id/  -    i1     ;    -r 


I 


«'.:).    str  in-s  >• 


Parameter  5  are 
spec  l  f  i  ed . 


called  by  value  unless  result  is 


Ulhen  events  appear  as  parameters,  the' •  are  active 
during  the  procedure  (unless  specifically  disabled!  .   If 
they  s>r^   signaled,  the  procedure  returns  immediately  and  th<i 
event  is  signaled  in  the  calling  routine. 

Examples: 

integer  mm,  max;   real  xi ; 
string  (10,1  wd;   on  back,  "escape" 
real  result  arret , peezet ,quzet 
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0.    ON  DECLrKriTION 


Jfj    string  \^          I    /                         ^T 
on   ~7T\.      / y>  \     \t .    statement  V* 

A    Vi.  id) •      \ 


Like  all  ether  declarations,  on  declarat ions  have 
static  scope. 

On  declarations,  also  called  event  indicators, 
indicate  a  course  of  action  to  be  taken  when  an  event 
included  in  the  list  is  signaled.   When  an  event  is 
signaled,  the  statement  associated  with  the  declaration  is 
execute:!.   Afterwards,  control  passes  to  the  statement 
following  the  declaration  unless  it  is  explicitly  transfered 
elsewhere. 

Several  event  identifiers  are  known  to  the  system: 

ans       data      help      labl      stop 
back      dotal     helpl      next 
back  I      formerror  lab        next  1 

When  the  user  presses  one  of  the  function  keys,  that  event 
is  signaled.   ElSECDSDCS:!!  ls  signaled  when  the  system  is  asked 
to  convert  a  string 'to  a  number  and  it  is  not  possible. 

The  term  key  is  active  whenever  a  string  literal 
■sfp^^rc    in  the  on  list.   When  the  user  presses  term  and 
enters  an  active  literal,  that  event  is  signaled. 

The  author  may  include  other  identifiers  in  the  event 
list.   These  are  activate:!  by  the  signal  statement.   (See 
pa  ge  b . ) 


Examples: 

[  on  help  hint: 

at  10  1t(;   accept: 

if  reply 

I  nearly  house  :   ok: 
I  else  :   no;   retru; 

]; 


i  *  I ; 

*  [  whi le  I  i^max  : 

[  if  names u) = name  : 

signal  found; 
]; 
i  *  1*1; 

]*; 

on  found  nr  >  nr*l; 
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APPENDIX  B 
INFORMATION  USED  FOR  SUBJECT  GROUPING 
Subjects  were  ranked  based  on  their  background  and  quiz  score  and  then 
assigned  to  a  language  group  according  to  a  predetermined  subject  grouping 
schema. 
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B. 1      Background   Information   Sheet 


BACKGROUND    INFORMATION  SHEET 

i 

Name: 

Social  Security  Number: 

What  is  y-:ur  current  status  at  the  University? 

1 .  Freshman 

2 .  Sophomore 

3.  Junior  . 

4.  Senior 

5.  Graduate  student  of  1  or   2   years 

6.  Graduate  student  of  3  or  more  yea? 


Indicate  which  ot   the  following  courses  for  equivalent)  you 
have  taken  or  sr<~   taking. 


1.  CS  121 

2.  CS  201 

3.  CS  311 
'4.  CS  32  3 

5.  CS  32? 

6.  CS  326 


Introduction  to  Computer  Programming 

Machine  Language  and  System  Drcgramming 

In  format  ion  Stru rtursf 

S>  'stem  F'r ':  5  rsmm  1  r.g 

Introduction  t  ;•  Programming  Languages 

Come  1 ler  Construction 


How  well  do  you  understand  the  following  languages? 

1.  never  heard  of  it 

2.  know  of  its  existance 

3..   know  a  little  a  I:  cut  it 

or  have  written  a  program  in  it 

4 .  know  it  f a  1  r  1  y  i\ie  1 1 

or  have  written  several  programs  in  it 

5.  know  practical  1"  everything  about  it 

or  have  programmed  extensively  using  it 


ALGOL  63:   1 

2 

3 

4 

5. 

ALGOLW:     1 

C 

3 

4 

5 

COBOL:      1 

2 

3 

4 

c 

EULER: 

2 

3 

4 

5 

FORTRAN: 

- 

3 

4 

5 

KAIL: 

*N 

3 

4 

e, 

LISP: 

"> 

3 

4 

r; 

PL'l: 

*> 

3 

4 

5 

SN0BC1. : 

1 

3 

4 

c; 

TUTOP: 

; 

3 

-4 

c 

5h 


Indicate  your  PLATO  experience. 

F 
I 

1.  I've  never  heard  of  PLmTO. 

2. 

3. 

4.  I've  used  PLATO  and  have  done  a  little  author  in-g. 

5. 

6. 

7.  I've  written  several  PLATO  lessons. 

3. 

* 

9. 

10.  I've  keen  working  e-tens ively  en  PLATO  for  more 
t Kan  f i ve  ye srs. 


Note:   The  numbers  which  are  not  labeled  -z;r^   meant  t  .->  be 
gradations  of  experience  in  between  th:>;e  which  are  labeled. 
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B.2     KAIL   Quiz 

KAIL   PRi.V.PrJI 

1  lesson    leapyr;  , 

2  '        ' 

3  procedure  introduction; 

4  erase; 

5  at  2813;   [  size  ~;   rotate  45  }wnte  LEn-  YErtR; 

6  at  3C27:   write  Press  NEXT;   pause  ne-t: 

7  end  introduction; 
8 

9  procedure  quest  ion; 

to  integer  yr; 

11  en  back  goto  int-c; 

12  er?se; 

13  at  101.0;   write  What  is  the  first  lesp  yes-  st'terl39V"; 

1  4  [  on  he  1  p  h  i  r.t ; 

I?  on  f-;rr,-ie*~r:t-  clean; 

16  at  12  1?;   (  long  6  }s:;ept  yr : 

17  if  yr 

13  I  1904  :   write  "Right,  •.••=•-■  good.; 

19  I  1902  :   no;   at  1417;   write 

23  ft    leap  yesr  must  be  divisible  by  f:u.r.; 

21  retry; 

22  I  i'JiiO  :   no;   at  I ■*!"";   write 

23  When. a  centesimal  •<■=?:-  is  invol^ezl. 

2  4  it  must  be  divisible  by  40  0.  ; 
2?  retry; 

26  I  else  :   write  "wrong,  tr1.-  sgam:   retry; 

27  ]  ; 

28  e1:    3027;   write  Press  NEXT;   pause; 

29  end  quest  1  ?n: 

3J  procedure  hint; 

31  at  1317;   I'irit'j 

32  A  lerp  year  is  ever-/  fourth  year  but  only 

33  those  centesimal  •••ear-  divisible  by  403.; 

3  4  pause;   clean; 
35  end  hint; 

36 

37  procedure  conclusion; 

35  on  back  goto  quest;   on  l::-:kl  goto  ir,tr:; 

39  erase; 

.   40  at  1217;   [  size  2.?  Jurite  END  OF  LESSON; 

41  p.?use  nevt , tack, back  1 ; 

42  end  conclusi :n; 
43 

44  intro.   ir.troduct  ion; 

4  5  quest.   question; 
-»6  con:  lusi  :>n; 
47 

43  end  leapyr: 
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r4«t£: 


QJJIZ 

SOCIrL   SECL'PITY  NUMBER: 


t*.         In   the  box,    reproduce    fapproximately)    the    :cntents  •:  f 
the  szreen   at    the    first   pause  after   the  prcgram  Ire-jins 

ex  scut  1  on .       i  5   p ■■:•  i  nt  si " 


2.  Assume  that  control  is  at  the  accept  statement  in  lin^ 
16.  What  is  the  first  mora  written  on  the  screen  often  th< 
student 

a.  presses  the  back  key'?  (3  points) 


b.  enters  "190>2"?  i3  points'! 


c.    enters    "1399'?     1 3    points) 


d.    presses   the   help    key?     (3   points) 
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3.    Assume  that  control  is  at;  the  ac.c.2Q£  statement  in  line 
16.   In  the  box,  reproduce  (approximately}  the  contents  cf 
the  screen  *t  tlie  nevt  pause  after  the  student  types  "19f0" 
and  ther.  p ;- ibres  the  help  key.  (S  points) 


-*.    fls5ur.ie  contro!  is    ?t  the  Eause  statement  in  lire  : 
What  is  the  first  word  writte-j  on  the  screen  ait-^r   the 
'student 

a.  presses  the  help  key?  (3  points) 


b.  presses  the  next  key?  (3  points) 


5. 


ab 


,/k 
/ 

lA 


L 


irxirr: 
>  i 


The  synta"-  diagram  ab-:-v-;  is  in  the  notation  given  ir 
r'riID'E.   Mhi-.h  :f  the  fo!  i  -.wins  sre  =yntact  l  ra.l  ly  :orrect 
according  to  this  diagram"  i"  points) 

\ 

a.  at  c.  adc, c,c  e.  atc.c.d.c.c 


b.  :-k: 


d.  oM 
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B. 3  Subject  Grouping  Schema 

A  function,  randu,  generated  random  choices  from  the  set  1,2.   A  1 
indicated  that  the  first  subject  of  a  pair  in  the  ranking  would  be  assigned 
to  group  T,  and  a  2  indicated  that  the  second  subject  would  be  assigned  to 
group  T. 

SUeJECT  GROL'Pir-'j  SIHErii 
x  =  randu  (2)  Tk         KT 


1 

2 

4 

■J 

6 

5 

7 

8 

9 

10 

12 

11 

13 

14 

16 

15 

17 

13 

20 

i9 

22 

21 

24 

23 

25 

26 

27 

26 

29 

30 

31 

32 

34 

33 

35 

36 

33 

37 

40 

39 
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APPENDIX  C 
COMPLETE  PROGRAM  LISTINGS  FOR  LESSON  "INTREP"  IN  T  AND  K 
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APPENDIX  D 
PERFORMANCE  STATISTICS  FOR  THE  DEBUGGING  SECTION 
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D.l  Scores 


Bug 

Me. 

ans 

df 

T 

K 

T 

26 

1 

5.8 

5-5 

.19 

2 

6.5 

6.1 

.25 

3 

5-2 

7.6 

2.16 

k 

U.9 

7.2 

1.39 

5 

U.2 

h.9 

.37 

6 

.8 

2.3 

i.6o 

7 

.U 

1.8 

1.27 

8 

.1 

.2 

.72 

Total 

27.8 

35-7 

1.18 

p (two-tailed) 


.oHo 


Table  Dl.  Scores:   Includes  All  Subjects  (13  K,  15  T) 


73 


Bug 

Number   of  ' 

Subjects 

Me 

:ans 

df 

T 

K 

T 

K 

T 

1 

13 

15 

5-8 

5.5 

26 

•  19 

2 

13 

111 

6.5 

6.6 

25 

•02 

3 

12 

Ik 

5.6 

8.1 

2k 

2.69 

k 

10 

13 

6.k 

8.3 

21 

1.27 

5 

8 

11 

6.9 

6.6 

IT 

.13 

T   p(two-tailed) 


013 


Table  D2 .   Scores:   Includes  Only  Subjects  Who  Worked  the  Problem 


Ik 


Bug      Number  of  Subjects         Means       df     T   p( two-tailed) 


K 

T 

K 

T 

1 

13 

15 

6.8 

6.7 

26 

.10 

2 

12 

Ik 

7.5 

7-6 

2k 

.07 

3 

10 

lli 

8.6 

11.2 

22 

1.52 

h 

8 

11 

11.3 

13.7 

17 

1.21 

5 

5 

8 

lU.O 

9-6 

11 

2.22 

0U9 


Table  D3.   Scores  Adjusted  to  Clve  More  Credit  to  Subjects 
Who  Performed  Well  Quickly 
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D.2  Timing  Data 

The  timing  data  consisted  of  the  following  variables: 

1.  time  spent  executing  "intrep" 

2.  time  spent  looking  at  the  listing  before  moving  forward 

3.  time  spent  looking  at  the  listing  before  searching  for  text 
h.      time  spent  looking  at  the  listing  before  moving  backward 

5.  time  spent  inserting 

6.  time  spent  replacing 

7.  time  spent  looking  at  the  listing  before  deleting 

8.  time  spent  on  miscellaneous  actions  including  time  spent  looking 
at  the  listing  before  inserting  and  replacing  and  before  accessing 
the  on-line  aids 

9.  time  spent  reading  the  reference  manual 

10.  time  spent  looking  at  the  flow  diagram 

11.  time  spent  obtaining  help  on  the  use  of  the  editor 

12.  sum  of  2,  3,  U,  and  8:   time  spent  looking  at  the  listing 

13.  sum  of  5,  6,  and  7:   time  spent  altering  the  listing 

lU.   sum  of  9»  10,  and  11:   time  spent  referencing  the  on-line  aids 
15.   total  time  spent 
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Variable  Means         df      T       p(two-tailed) 

1 
2 

3  29.0     ii.6         1.86         .07^ 

k 

5 

6  9U.2     28.3        2.89        .008 


7 
8 

9 
10 
11 
12 
13 
lU 
15 


Me; 

ans 

df 

T 

K 

T 

26 

11U.8 

119.9 

.10 

77-2 

66.3 

.k2 

29.0 

11.6 

1.86 

9-9 

17.5 

.87 

6.8 

25.3 

1.26 

9U.2 

28.3 

2.89 

.0 

3.5 

71.2 

80.3 

.36 

11.1 

11.6 

.05 

.8 

.0 

18.0 

15-2 

,2U 

187.  k 

175-7 

•  23 

101.1 

57-1 

1.70 

29.8 

26.8 

.17 

U33.2 

379.5 

.52 

Table   Dh.      Timing  Data   for   Bug  1    (13  K,    15  T] 
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Variable 

Me 

ans 

df 

T 

p(two-tailed) 

K 

T 

2k 

1 

115.  k 

109.6 

.38 

2 

125-7 

105.9 

• 

•  52 

3 

70.3 

6l.O 

.U2 

k 

21.1 

31.1 

.88 

5 

.0 

18. 7 

r 

6 

.6 

3.U 

1.28 

7 

7.5 

.0 

8 

88.8 

70.9 

•  90 

9 

5-8 

7.8 

.28 

10 

3.1 

.8 

1.06 

11 

7-6 

k.6 

•  70 

12 

305-8 

268.9 

•  75 

13 

8.3 

22.1 

2.83 

.009 

Ik 

16.5 

13.1 

•  33 

15 

UU6.0 

1+13.8 

-53 

Table  D5-   Timing  Data  for  Bug  2  (12  K,  Ik   T) 
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Variable 

Me 

ans 

df 

T 

K 

T 

22 

1 

UU.9 

^5-7 

.07 

2 

31.3 

27.1+ 

.33 

3 

6.7 

6.8 

.01 

1+ 

13.6 

25.0 

1.07 

5 

39-^ 

22.6 

2.33 

6 

U.9 

9.6 

•  76 

7 

.0 

.h 

3 

1+2.0 

35.0 

.85 

9 

2.7 

Ik. 6 

1.23 

10 

.0 

.0 

11 

1.1 

.0 

12 

93.6 

9U.2 

.02 

13 

UU.3 

32.6 

1.13 

lk 

3.8 

1U.6 

1.11 

15 

186.6 

187.1 

.01 

p( two-tailed) 


030 


Table  D6.   Timing  Data  for  Bug  3  (10  K,  lk   T) 
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Variable  Means  df  T       p(two-tailed) 

K  T  IT 

1  36.8  30.7  .70 

2  30.1  35.1  .U8 

3  13.9  7-0  .78 

1+  6.9  .8  1.89           .080 

5  16.9  11.1  1.65 

r 

6  3.5  2.9  .19 

7  3.0  .0 

8  k2.k  27-8  1.39 

9  .0  1.5  .85 

10  .0  .6 

11  -0  .0 

12  93.3  70.7  1.31* 

13  23. 1+  1U.0  1.66 
lU  .0  2.1 

15  153. ^  H7.5  i.hk 


Table  D7 .   Timing  Data  for  Bug  h    (8  K,  11  T) 
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Variable 

Me 

ans 

df 

T 

p(two-tailed) 

K 

T 

11 

1 

55-0 

57-9 

.23 

2 

32.  k 

66.U 

2.87 

.015 

3 

12.0 

7-U 

•  71 

1+ 

9-0 

12.8 

•  32 

5 

9.U 

10.3 

.15 

6 

2.U 

2.6 

.08 

7 

.0 

3.5 

8 

23.8 

30.0 

.6k 

9 

.0 

.5 

10 

.0 

.0 

11 

.0 

1.3 

12 

77. 2 

116.5 

1.92 

.081 

13 

11.8 

16.1+ 

.63 

lU 

.0 

1.8 

15 

lUU.o 

192.0 

1.51 

Table  D8.   Timing  Data  for  Bug  5  (5  K,  8  T) 
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Variable 

Me 

ans 

df 

T 

p( two-tailed) 

K 

T 

22 

1 

355-1 

337-8 

•  50 

2 

326.9 

313.9 

•  25 

3 

136.5 

117.8 

.U6 

k 

U8.8 

102.7 

3.U3 

.002 

5 

55-9 

86.2 

1.89 

.072 

r 

6 

109.  h 

58.8 

2.03 

.056 

7 

9-1 

5-7 

•  lh 

8 

253.1 

259.1 

•  3U 

9 

31.2 

38.6 

.3U 

10 

3.9 

1.5 

•  99 

11 

30.3 

18.8 

•  77 

12 

765. 3 

739-5 

.60 

13 

17U.1* 

150.7 

•  9U 

Ik 

65-3 

58.8 

.26 

15 

1360.2 

13^0.8 

1.U6 

Table  D9.   Timing  Data  Totals  for  All  Subjects  Who 
Completed  the  Debugging  Section 
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D.3  Editing  Data 

The  editing  data  consisted  of  the  following  variables: 

1.  number  of  executions  of  "intrep" 

2.  number  of  forwards,  F 

3.  number  of  searches,  X 
h.  number  of  backs,  B 

5.  number  of  inserts,  I 

6.  number  of  replaces,  R 

7.  number  of  deletes,  D 

8.  number  of  editing  keys  incorrectly  entered  or  partially  entered 
and  then  erased 

9-  number  of  accesses  to  the  reference  manual 

10.  number  of  references  to  the  flow  diagram 

11.  number  of  requests  for  help  in  editing 

12.  sum  of  2,  3,  and  h :      number  of  searching  options  activated 

13.  sum  of  5,  6,  and  7:   number  of  alterations 

lU.  sum  of  9 ■>    10,  and  11:   number  of  references  to  the  on-line  aids 

15.  sum  of  1,  12,  13,  and  lU:   total  editing  keypresses 
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Variable  Means         df      T       p(tvo-tailed) 

K  T      26 

1  1.6  1.9            -87 

2  k.h  U.T           .19 

3  1.9  -8  1.61* 
h  .8  1.1*  1.03 

5  .2  .7  1.63 

6  l.i  .9          .1*3 

7  .0  .1* 

8  2.9  3.2           .29 

9  .2  .2           .30 

10  .8  .0 

11  .8  .7           .23 

12  7-0  6.9           .03 

13  1.2  2.1  1.39 
lU  1.0  .9  .27 
15  10.8  11.8           .29 


Table  D10.   Editing  Data  for  Bug  1  (13  K,  15  T] 
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Variable  Means                      df  T                 p( two-tailed) 

K  T              2k 

1  1.5  2.1  1.1+0 

2  8.7  7-5  .36 

3  5-1  U.l  -70 
U  1.6  2.1+  .89 

5  .0  .9 

6  .1  .2  .90 

7  .5  .0 

8  1+.7  3.1  1.1+1 

9  .2  .1  .16 

10  .3  .1  -98 

11  .7  .U  .79 

12  15.3  lU.O  .35 

13  .7  l.l  1.8U                       .078 
ll+  1.1  .6  1.00 

15  18.6  17.8  .19 


Table  Dll.   Editing  Data  for  Bug  2  (12  K,  lU  T) 
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Variable  Means                      df  T                 p( two-tailed) 

K  T  22 

1  1.3  l.lt  .23 

2  1.8  2.1  .1+6 

3  .7                   -9  .22 
1+  l.O  2.1  1.27 

5  1.1  1.2  .56 

6  .2  .1+  .98 

7  .0  .1 

8  1.9  1.8  .31 

9  .1  .2  .72 

10  .0  .0 

11  .2  .0 

12  3-5  5.1  .90 

13  1.3  1-7  1.21 
lit  .3  .2  .39 
15  6.1+  8.U  .96 


Table  D12.      Editing  Data  for  Bug   3    (10  K,   lit  T] 
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Variable 

Means 

df 

T 

p (two-tailed) 

K 

T 

IT 

i 

1.3 

1.2 

.3k 

2 

2.0 

2.2 

.38 

3 

1.1 

.1+ 

1.1+6 

1* 

•  5 

.1 

1.61+ 

5 

1.3 

.8 

2.16 

.01+5 

6 

.3 

• 

.2 

.3U 

/■ 

7 

.2 

.0 

8 

1.6 

1.9 

9 

.0 

.1 

10 

.0 

.1 

11 

.0 

.0 

12 

3.6 

2.6 

2.71 

.015 

13 

1.8 

1.0 

lli 

.0 

.2 

15 

6.6 

5.0 

2.23 

.oi+o 

Table  D13.      Editing  Data  for  Bug  1+    (8  K,   11  T) 
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Vari 

able 

Means 
K 

T 

df 
11 

T 

p(  two-tailed.) 

1 

1.2 

1.5 

.67 

2 

3.6 

h.9 

l.lU 

3 

1.1* 

•  9 

.72 

k 

.6 

1.3 

.63 

5 

1.0 

1.0 

6 

.2 

.3 

.19 

7 

.0 

•  5 

8 

1.2 

2.5 

9 

.0 

.1 

10 

.0 

.0 

11 

.0 

.0 

12 

5.6 

7.1 

.81+ 

13 

1.2 

1.7 

.68 

lit 

.0 

.3 

15 

8.0 

10.6 

1.08 

Table  DlU.   Editing  Data  for  Bug  5  (5  K,  8  T) 


Variable  Means         df  T       p(two-tailed! 

K  T      22 

1  6.7  9.2  3.38         .003 

2  22.3  23.2  .18 

3  9.6  9.U  .05 

k  k.l  8.3  3.06          .006 

5  2.1+  U.5  2.9^          .00? 

6  2.8  2.6  .19 

7  .7  .7  .00 

8  11.2  13.2  1.27 

9  .6  .9  .73 

10  .3  .2  .76 

11  1.8  l.l  .98 

12  36.O  U0.9  1.05 

13  5.8  7.8  1.53 
Ik  2.6  2.5  .58 

15  51.2  60.0  1.73        .098 


Table  D15.   Editing  Data  Totals  for  All  Subjects  Who  Completed 
the  Debugging  Section 
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APPENDIX  E 
PERFORMANCE  STATISTICS  FOR  THE  MODIFYING  SECTION 
E.l  Scores 

The  modifying  section  of  the  experiment  was  scored  as  follows: 

1.  Specification  #1 

a.  Control  Linkage  (15  points) 

b.  Procedure  (25  points) 

2.  Specification  #2 

a.   Control  Linkage  (35  points) 

"b.   Procedure  (25  points) 
Two  K  subjects  completed  specification  1,  and  then  inadvertently  exited  the 
experiment. 


90 


Item  Scored  Means  df  T  p (two-tailed) 


K 

T 

Control  Linkage  1 

11.9 

lU.i 

23 

1.3U 

Procedure  1 

16.8 

18.8 

23 

•  97 

Specification  1 

28.7 

32.8 

23 

1.U6 

Control  Linkage  2 

9-1 

8.8 

21 

.13 

Procedure  2 

19.  k 

15-2 

21 

1.96 

Specification  2 

28.5 

2U.1 

21 

1.30 

Total  Performance 

56.0 

56.9 

21 

.16 

063 


Table  El.   Scores  for  the  Modifying  Section 
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E.2  Timing  Data 

The  timing  data  consisted  of  the  following  variables: 

1.  time  spent  reading  the  specifications  initially 

2.  subsequent  time  spent  reading  the  specifications 

3.  time  spent  looking  at  the  listing  before  moving  forward 

h.      time  spent  looking  at  the  listing  before  searching  for  text 

5.  time  spent  looking  at  the  listing  before  moving  backward 

6.  time  spent  inserting 

7.  time  spent  replacing 

8.  time  spent  looking  at  the  listing  before  deleting 

9-   time  spent  on  miscellaneous  actions  including  time  spent  looking 
at  the  listing  before  inserting  and  replacing  and  before  accessing 
the  on-line  aids 

10.  time  spent  reading  the  reference  manual 

11.  time  spent  looking  at  the  flow  diagram 

12.  time  spent  obtaining  help  on  the  use  of  the  editor 

13.  sum  of  1  and  2:   time  spent  reading  the  specifications 

ik .      sum  of  3,  i+,  5,  and  9-      time  spent  looking  at  the  listing 
15-   sum  of  6,  7,  and  8:   time  spent  altering  the  listing 

16.  sum  of  10,  11,  and  12:   time  spent  referencing  the  on-line  aids 

17.  total  time  spent  excluding  the  time  spent  reading  the  specifications 
initially 
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Variable 

Me 

ans 

df 

T 

p(two-tailed) 

K 

T 

21 

1 

100.0 

106.8 

•  51 

2 

135.0 

10U.9 

1.03 

3 

151-5 

1U8.5 

.06 

h 

68.0 

12U.2 

1.U9 

5 

97-9 

108.2 

.30 

6 

U23.6 

379.8 

.93 

7 

107.3 

J47.3 

2.58 

.017 

8 

6.U 

12.8 

1.23 

9 

283-7 

271.8 

•  3U 

10 

U8.0 

98.8 

1.30 

11 

l.T 

U.3 

1.10 

12 

11.  U 

5.0 

1.31 

13 

235-0 

211.8 

•  72 

Ik 

601.1 

652.6 

.75 

-    15 

537-3 

U39.8 

2.01 

.058 

16 

61.1 

108.2 

1.16 

IT 

li+3^.5 

1U12.I4 

.28 

Table  E2.   Timing  Data  for  the  Modifying  Section 
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E.3  Editing  Data 

The  editing  data  consisted  of  the  following  variables: 

1.  number  of  references  to  the  specifications 

2.  number  of  forwards,  F 

3.  number  of  searches,  X 
h.      number  of  backs,  B 

5.  number  of  inserts,  I 

6.  number  of  replaces,  R 

7.  number  of  delets,  D 

8.  number  of  editing  keys  incorrectly  entered  or  partially  entered 
and  then  erased 

9.  number  of  accesses  to  the  reference  manual 

10.  number  of  references  to  the  flow  diagram 

11.  number  of  requests  for  help  in  editing 

12.  sum  of  2,  3,  and  h:    number  of  searching  options  activated 

13.  sum  of  5,  6,  and  7:  number  of  alterations 

li+.   sum  of  9,  10,  and  11:  number  of  references  to  the  on-line  aids 
15-   sum  of  1,  12,  13,  and  lU :   total  editing  keypresses 
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Variable 

Means 

df 

T 

p(two-tailed) 

K 

T 

21 

1 

3.6 

3.5 

.21 

2 

10.5 

12.2 

•  37 

3 

5-2 

11.1 

1.75 

.095 

1+ 

9.2 

10.0 

•  32 

5 

6.9 

9-3 

2.65 

.015 

r 

6 

3.3 

2.7 

.1+5 

7 

.7 

1.5 

1.25 

8 

9.1 

10.1 

.61+ 

9 

.8 

1.8 

1.87 

.076 

10 

.1 

.U 

1.55 

11 

•  9 

.h 

1.1+0 

12 

2I+.9 

33.2 

1.58 

13 

10.9 

13.7 

1.93 

.068 

Ik 

1.8 

2.5 

1.03 

15 

1+1.2 

52.9 

2.10 

.01+8 

Table  E3.   Editing  Data  for  the  Modifying  Section 


APPENDIX  F 
CORRELATIONS 
A  missing  data  correlation  program  was  used  to  compute  these  statistics. 
Correlation  factors: 

1.  University  status 

2.  Courses 

3.  Languages 

k.      PLATO  experience 
5-   KAIL  quiz  score 

6.  Rank 

7.  Score  on  debugging  section 

8.  Time  spent  in  "intrep" 

9.  Time  spent  looking  at  listing  (debugging) 

10.  Time  spent  altering  listing  (debugging) 

11.  Total  editing  keypresses  (debugging) 

12.  Score  on  modifying  section 

13.  Time  spent  reading  specifications 
Ik.      Time  spent  looking  at  listing  (modifying) 

15.  Time  spent  altering  lsiting  (modifying) 

16.  Total  editing  keypresses  (modifying) 

For  correlation  factors  1  through  6,  13  K  and  15  T  subjects  were  considered; 
for  correlation  factors  7  through  11,  12  K  and  12  T  subjects  were  considered; 
and  for  correlation  factors  12  through  l6 ,  10  K  subjects  and  13  T  subjects 
were   considered.   In  the  following  tables 

*  =  significant  with  probability  .05,  and 
**  =  significant  with  probability  .01. 
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